Cross-Corpus Evaluation of Word Alignment

We present the procedures we implemented to carry out system oriented evaluation of a syntax-based word aligner —ALIBI. We take the approach of regarding cross-corpus evaluation as part of system oriented evaluation assuming that corpus type may impact alignment performance. We test our system on three English–French parallel corpora. The evaluation procedures include the creation of a reference set with multiple annotations of the same data for each corpus, the assessment of inter-annotator agreement rates and an analysis of the reference sets. We show that alignment performance varies across corpora according to the multiple references produced and further motivate our choice of preserving all reference annotations without solving disagreements between annotators.